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What Is Predictive Modeling? 

Predictive modeling estimates individuals’ future outcomes or their probabilities of future 
outcomes. It does this by building and testing a model using data on similar individuals whose 
outcomes are already known. Commonly used in business and marketing research, predictive 
modeling is gaining currency in many social policy domains as a way to identify individuals 
who may benefit from targeted intervention. When results are interpreted correctly, predictive 
modeling offers benefits to those engaged in continuous improvement efforts and those who are 
looking to allocate resources more efficiently. 

Why Use Predictive Modeling in Education? 

Schools and school districts often use early warning systems to identify students who are at risk 
of not meeting key academic outcomes, such as graduating on time or passing state exams . 1 
Early warning systems typically use student-level data to generate a limited set of indicators to 
classify whether students are at risk. For example, the risk of not graduating from high school is 
commonly estimated by the so-called ABC indicators of attendance, behavior, and course 
perfonnance . 2 An indicator-based approach typically produces a binary measure (at risk or not 
at risk) or a categorical measure of risk (for example, low, medium, or high) based on a snap- 
shot of readily available measures of student behavior and perfonnance. 

Yet education systems are increasingly creating rich, longitudinal data sets with fre- 
quent, and even real-time, data updates of many student measures, including daily attendance, 
homework submissions, and exam scores. These data sets provide an opportunity for district 

Tor example, see Frazelle and Nagel (2015); George Washington University (2012); Stuit et al. (2016); 
Therriault et al. (2010); Therriault et al. (2013). 

“Allensworth and Easton (2007); Balfanz, Herzog, and Mac Iver (2007); Balfanz, Wang, and Byrnes 
(2010); Celio (2009a, 2009b); Frazelle and Nagel (2015); Mac Iver (2010); Mac Iver and Mac Iver (2009); 
Roderick (1993); Uekawa, Merola, Fernandez, and Porowski (2010). 


1 



and school staff members to move beyond an indicators-based approach and instead employ 
new methods to compute more frequent, more accurate, and more nuanced predictions of 
student risk. 

For these reasons, MDRC has been exploring the value of predictive modeling. MDRC 
researchers have developed and implemented a comprehensive predictive modeling framework 
that allows for rapid and iterative estimation of a continuous measure of risk (a probability 
between 0 and 1 of not achieving an outcome) for each student at a point in time. The frame- 
work was developed during a partnership with New Visions for Public Schools (NVPS), a 
nonprofit organization that works with more than 200 public schools in New York City. The 
framework has so far been implemented with data from its network of 70 high schools to 
estimate students’ risk of not graduating on time and of not passing the state algebra exam 
required for graduation. 

Consider an example in which midway through ninth grade, district administrators and 
educators want to know each student’s risk of not graduating from high school on time. With the 
common ABC indicators approach, a student with a low attendance rate, behavior incidents, or 
course failures may be designated as having moderate or high risk of not graduating at the end of 
twelfth grade, depending on how the student’s first-semester outcomes (and possibly eighth-grade 
outcomes) compare with district-specified thresholds of these ABC indicator measures. 

With a predictive modeling approach, instead of a parsimonious set of summary 
measures (the indicators), hundreds of measures can be considered for determining students’ 
risk. Not only students’ overall first-semester attendance rates but their attendance pattern can 
be considered — good attendance on average could be masking a drop-off later in the semester 
or a large gap at some point. Nonacademic measures, such as involvement in the child welfare 
or justice system, if available, can also be taken into account. The infomiation from a large 
number of measures can be extracted with machine learning (though sometimes a simple 
regression model may work just as well) and summarized in a single, continuous estimate of 
risk. Instead of a student being designated as at moderate or high risk, she is determined to have, 
for example, a 70 percent chance of not graduating on time, plus or minus 5 percentage points 
(with values closer to 70 percent being more likely). 

Such estimates allow students to be ranked by their risk levels, and thresholds can be 
determined for particular interventions. Figure 1 illustrates results that could be presented for a 
hypothetical school. It shows each student’s predicted probability (indicated by a circle) of not 
achieving a particular milestone, such as on-time graduation from high school. The figure also 
shows the estimated uncertainty interval around each prediction (the vertical line). The students 
have been ranked by their predicted probabilities. Such a plot reveals the variation across 
students in the school and the risk levels around which students may cluster. 
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Predicted probability 


Figure 1. Predicted Probabilities of Not Achieving a Milestone and Estimated Uncertainty Intervals, 
by Student at a Hypothetical School, Ranked by Probability 



Looking at the distributions of risk across and within schools may provide new insights 
about variation in students’ needs. For example, among those students classified as “high risk” 
by an indicators approach, the estimates from predictive modeling may reveal substantial 
variation in risk levels. This can be combined with descriptive analyses of students’ academic 
progress within different ranges of risk to reveal particular factors that correlate with risk level. 
Finally, at the end of ninth grade, all students’ risk estimates can be updated with new infor- 
mation — both the model and the predictions are updated. Administrators and educators can 
then track students’ changes in risk on dashboards or other communication systems. 
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What Is MDRC’s Approach to Predictive Modeling? 

MDRC’s approach to predictive modeling of student risk has the following features: 

• It relies on close partnership with practitioners. Because policy and prac- 
tice can change during and between school years, researchers need to work 
with practitioners to deepen their understanding of how to use and create 
measures from existing data sets, interpret results correctly, and maximize 
the usefulness of results. 

• It produces results that capture important variation between students in 
levels of risk and allows for students to be ranked by their risk estimates. 

A predicted likelihood between 0 and 1 of not achieving an outcome pro- 
vides more infomiation about a student’s risk than a category can. Also, as- 
sessing students by a continuous measure of risk better allows administrators 
to rank them in priority for different types of interventions. 

• It uses an analytic framework with specified decision rules and field- 
tested statistical code. The framework includes analytic steps focused on 
(1) identifying the best samples for training the statistical model and compu- 
ting predictions; (2) processing data (for example, handling missing values, 
creating aggregate measures that extract useful infonnation such as changes 
in daily attendance, and identifying and creating measures that have the same 
meaning over time); (3) selecting measures to include in modeling (relying 
on a combination of substantive knowledge and data-adaptive algorithms); 
(4) identifying the best modeling methodology; (5) estimating uncertainty in 
predictions; and (6) summarizing and interpreting results. 

• It extracts as much information as possible from data, allowing for hun- 
dreds of measures to be considered as potential predictors in modeling. 

In MDRC’s framework, the modeling incorporates both standard regression- 
based approaches and machine learning algorithms, which can search very 
large numbers of measures and let the data determine what fonn a statistical 
model should take. MDRC’s approach compares multiple models and selects 
the best one based on its predictive performance in new samples (that is, data 
that were not used to train the model; this helps avoid overfitting, in which 
the model does not adequately capture underlying relationships in the data 
and therefore does not generalize well to new data). These comparisons are 
done with cross-validation, a data resampling method that mimics repeatedly 
fitting a model in one sample and then evaluating it in a different sample. 
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When researchers consider multiple models or multiple machine learning al- 
gorithms, it is known as “ ensemble learning .” The way that MDRC research- 
ers employ ensemble learning draws on both substantive expertise — 
through the parametric models specified by researchers based on their 
knowledge of which predictors matter — and the data-adaptive techniques 
used by machine learning algorithms. The researchers assess predictive per- 
formance in a variety of ways, emphasizing model performance measures 
that align with partners’ priorities for how the results will be used. For exam- 
ple, by focusing on the metrics of sensitivity and specificity , MDRC aims to 
maximize identification of truly at-risk students while minimizing false 
alarms. 

• It allows for rapid iteration and replication. As described above, MDRC 
has developed a repeatable multistep framework that allows one to update 
predictive models as new infomiation becomes available and to easily repli- 
cate the process to predict additional milestones. The ability to rapidly iterate 
is an advantage over indicator-based systems. Districts creating such systems 
tend to combine indicators from published literature and descriptive analyses 
and use subjective decisions linked to district priorities to determine the final 
set of indicators. Such an approach makes updating time-consuming, espe- 
cially when changing the outcome to a different milestone (for example, 
from failure to graduate to course failure). 

• It estimates uncertainty in prediction results. As with any statistical pro- 
cedure, the estimates of students’ likelihoods of not achieving milestones 
have uncertainty, which is typically ignored in indicator-based approaches 
and even other predictive modeling frameworks. MDRC uses a nonparamet- 
ric bootstrap procedure to estimate uncertainty and provide lower and upper 
bounds of predicted likelihoods. The nonparametric bootstrap procedure 
resamples data in order to mimic repeated draws from a population, and the 
entire analytic process can be repeated in each draw so that variation across 
samples can be estimated. 

• It guides practitioners in maximizing use of results. After producing indi- 
vidual predictions of whether each student will fail to meet a particular mile- 
stone, and estimates of uncertainty in those predictions, MDRC works with 
partners to provide post-hoc analyses of predicted likelihoods that summarize 
variation across schools and subgroups of students. MDRC also guides prac- 
titioners in understanding the value and limitations of the predictions and in 
communicating results to school leaders and teachers. 
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What Are MDRC’s Future Directions in Predictive Modeling of 
Academic Outcomes? 

Researchers at MDRC are investigating ways to improve its predictive modeling of student risk. 
They are investigating whether different types of data — both from the education system and 
beyond — could improve the accuracy of predictions, especially predictions of long-term 
outcomes made early in students’ careers. In particular, they are investigating whether nonaca- 
demic data, such as students’ receipt of social services, or additional academic data that districts 
are increasingly collecting, such as homework completion and other course-level metrics, can 
improve predictive models for a variety of outcomes. MDRC researchers are also working to 
make iteration of the framework more efficient, by streamlining data processing steps and 
speeding up computing. In addition, MDRC is investigating methodological questions such as 
the extent to which and under what circumstances machine learning adds value, as well as 
which machine learning algorithms might be most valuable. 

MDRC can support district partners in several ways: First, its researchers can help as- 
sess whether districts have the organizational capacity and the data to support predictive 
modeling and whether it is worthwhile for their particular needs. Where predictive modeling is 
suitable, MDRC researchers can help districts make the most of the results to guide decision- 
making. Districts interested in this approach might benefit from incorporating the results — 
both individual predictions and summaries across students, grades, and schools — into their 
dashboards and other data visualization and data sharing systems. MDRC can also provide 
analyses to help educators decide what to do next — how best to intervene and for whom. 


Definitions of Technical Terms 

Machine learning: Machine learning refers to methods that automate model building using 
iterative algorithms — that is, a series of steps that continuously adjust for better predictive 
perfonnance — rather than relying on functional forms (specifications of the independent 
variables and the types of relationships between the variables and the outcome) specified by an 
analyst. Examples of machine learning algorithms that MDRC researchers have employed 
include decision frees, random forests, stepwise regression, k-nearest neighbors, support vector 
machines, and others. 

Cross-validation: The MDRC framework uses v-fold cross-validation in which the data with 
known outcomes (for example, whether students graduated or not) are partitioned into a certain 
number (v) of subsamples (or folds). A common value for v is 5, but other values are used too, 
often driven by the sample size. A model is fit in all but one of the folds, and measures of 
perfonnance are computed in the remaining, “validation” fold. The process is repeated v times 
such that each fold takes a turn as the validation fold. The averages of the perfomiance 
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measures are computed across all validation folds. The entire process can be repeated multiple 
times in order to reduce the variance of the cross- validated estimates. 

Ensemble learning: Ensemble learning allows an analyst to consider multiple models or 
machine learning algorithms in a systematic and prespecified way so that the optimal model or 
algorithm, according to a performance measure of interest, is selected and used for prediction. 

Sensitivity and specificity: When the outcome of interest is failing to achieve a milestone, 
sensitivity is the proportion of those who actually fail to achieve it who are correctly predicted 
as failing (based on a predetermined threshold in estimated continuous likelihood to fail). The 
specificity is the proportion of those who actually do achieve the milestone who are correctly 
predicted as doing so. The false positive rate is equal to 1 - specificity. Often it may make sense 
to find a model that makes the best trade-offs between sensitivity and specificity. Plotting and 
computing the area under receiver operating characteristic (ROC) curves is a useful way to 
summarize the trade-offs across all possible thresholds for designating at-risk students. 

Nonparametric bootstrap: With a nonparametric bootstrap, the data are resampled with 
replacement (that is, observations in the first draw are eligible for selection in subsequent 
draws) a large number of times (say, 1 ,000), and analyses are repeated within each resample. 
The resulting set of 1 ,000 estimates provides an approximation of the sampling distribution of 
the estimators of interest. When MDRC researchers apply the nonparametric bootstrap to 
predictive modeling, it produces 1,000 predicted likelihoods for each student. The percentiles 
from 2.5 to 97.5 provide a range with a 95 percent chance of containing a student’s true likeli- 
hood, with values closer to the center of the distribution being more likely. 


For more information, contact: 
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